Task Pipeline Specification and Scheduling

Research pipelines, such as those often found in computer vision or computational biology, often consist of a large number of heterogenous programs. This leads to brittle code that is difficult to maintain, while requiring significant effort to parallelize across multiple machines. We present a lightweight framework for specifying the execution of such pipelines. The structure imposed by our framework allows us to optimize the execution of the pipeline. Specifically, we know what resources (i.e. files) each task in the pipeline depends on and produces. We can use this information to intelligently assign jobs to different machines in the cluster. Our framework executes pipelines on clusters of machines with minimal effort (and automatic dependency packaging) while scheduling tasks to minimize time until completion (including file transfer time). We compare different heuristics used to initialize our algorithm, and show that the scheduling algorithm used can yield a significant impact on performance. We also give results on running a real-world pipeline with a single machine and multiple machines. Lastly, we compare the amount of code required to implement similar pipelines using different frameworks.